Skip to content

feat: durable message lifecycle from admission to execution - #3721

Open
Astro-Han wants to merge 24 commits into
apache:mainfrom
Astro-Han:fix/durable-message-lifecycle
Open

feat: durable message lifecycle from admission to execution#3721
Astro-Han wants to merge 24 commits into
apache:mainfrom
Astro-Han:fix/durable-message-lifecycle

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

This is the first end-to-end durable message lifecycle PR built from main, replacing Draft PR #3633 without cherry-picking or continuing its patch series.

The contract is simple: once turn.message.submit accepts a message, a Host crash must not lose it, revive it after settlement, or execute it twice.

submit
  |
  v
Accepted (draft; admission authority only)
  +-- retract / Stop before handoff --> Cancelled (no history)
  |
  +-- durable root-source or steering proof
        |
        v
      atomic transcript materialization + HandedOff
        +-- terminal Stop -------------> Cancelled (history retained)
        |
        +-- downstream provider-request proof --> Executed

These are durable classifications, not an independent provider state machine. queued, leased, and in_flight remain reconstructible Host-memory projections.

First-principles ownership

  • message_admissions is the sole durable authority for an accepted draft, including canonical content, edit, reorder, promotion, retract, and lifecycle classification.
  • Accepted queued messages do not appear in transcript or catalog preview. Retracting one therefore only settles it as Cancelled; no append-only history must be deleted or repaired.
  • RootAdmissionOwner owns the durable Root execution contract and source-message proof.
  • One SQLite handoff operation materializes canonical transcript rows in final execution order and advances Accepted -> HandedOff in the same transaction.
  • An immediately admitted root already has transcript identity from the atomic Root admission composition; handoff validates that exact row and advances lifecycle without creating a duplicate.
  • A multi-message successor has userMessageId = null and keeps its individual sourceMessages; Runtime does not invent an aggregate transcript message.
  • HostMessageCoordinator owns only the reconstructible queue projection and the shared proof classifier/settlement path. Normal completion and restart recovery call the same owner.
  • Executed requires a provider-request proof downstream of the admitted Root contract.
  • Durable Session capability binding is derived from the durable Root execution contract, not a restart-volatile marker.
  • RuntimeKernel no longer owns queue state, leasing, folding, retract, fallback authority, or a second root-message materialization path.

Crash-cut behavior

  1. Before canonical admission: no accepted message or transcript row exists.
  2. After queued admission, before handoff proof: exactly one Accepted draft exists and transcript has zero rows for it.
  3. After Root admission, before activation: the durable Root contract proves exact source identities; handoff materializes each source once under the target Turn.
  4. After handoff: recovery sees HandedOff plus its transcript identity and never puts it back in the queue.
  5. After a downstream provider request: recovery settles Executed and never replays it.
  6. A retract or interrupt before handoff settles Cancelled without history. Terminal cancellation after handoff retains history.

Scope

Included: atomic admission/root transcript composition, canonical Skill admission, edit/reorder/promotion/retract, ordered successor handoff, proof-driven settlement, restart recovery, size boundaries, durable capability derivation, and removal of old Runtime queue/transcript authority.

Excluded: CLI/Desktop projection refactors and Side Conversation UI/hooks.

Verification

Only affected tests were run locally; the full repository suite was not run locally.

  • Exact pushed head: 48b162326.
  • Builds passed: @maka/storage and @maka/runtime-host.
  • SQLite metadata tests: 53 passed.
  • SQLite SessionStore tests: 22 passed.
  • Root coordinator tests: 50 passed.
  • Canonical projection, message coordinator, real Runtime Host/UDS message and queue suites passed.
  • Fault coverage includes canonical Skill failure before Root admission, real SQLite reorder plus restart, Accepted-draft retract and interrupt with zero transcript rows, queue-admission Host crash, admitted Root without a Run, ordered multi-source successor handoff, immediate-root transcript validation, and transcript de-duplication.
  • Exact-head CI run 32806065640 passed, including Runtime Host tests and installed CLI release-candidate validation.

Review follow-up

  • Fixed real reorder permutations by validating identity membership and persisting the requested order transactionally.
  • Removed the doubled raw/prepared admission envelope; durable admission stores one canonical content value plus the submitted digest.
  • Moved Skill preparation and rejection ahead of admission, closing raw /skill replay and rejected phantom-history paths.
  • Removed eager Accepted transcript/catalog mirroring and the separate transcript-rebind operation.
  • Collapsed transcript materialization and HandedOff into one SQLite transaction owned by the proof-driven lifecycle settlement path.

AI use

This PR was implemented with Codex assistance. The design, repository decisions, code changes, affected-test selection, review of Draft PR #3633 as evidence, adversarial review, simplification audit, and verification were directed and checked against the repository's durable authorities and production composition.

@Astro-Han
Astro-Han marked this pull request as ready for review August 24, 2026 18:21

@M4n5ter M4n5ter left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

English

I found two correctness issues on exact head 2ea21a68950502615f5e109ce914a7af94a97140: one makes durable follow-up reordering fail for every real permutation, and the other rejects otherwise valid 32–49 KiB messages as an internal failure. The exact-head hosted test check is green, but its queue test uses an in-memory lifecycle stub for reorder and does not exercise either storage boundary.

简体中文

我在 exact head 2ea21a68950502615f5e109ce914a7af94a97140 上确认了两个正确性问题:持久化 follow-up 队列的任何实际换序都会失败;另一个问题会把本来合法的 32–49 KiB 消息错误地变成内部错误。当前 head 的托管 test 检查是绿色,但队列测试对重排使用了内存 lifecycle stub,没有覆盖这两个真实存储边界。

Comment thread packages/storage/src/sqlite-session-metadata-store.ts Outdated
Comment thread packages/runtime-host/src/server/message-coordinator.ts

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this head and found blocking issues that need to be addressed before merge.

[P1] Recovery can replay raw /skill text instead of the prepared skill invocation

Idle submit persists raw modelContent first (message-coordinator.ts:906-918) and only later expands the skill in the Host (root-turn-coordinator.ts:1028-1077). If the Host exits between those steps, recovery replays the persisted raw content without re-running skill preparation, producing a root that the normal path would have wrapped.

[P1] Rejected idle submit leaves a phantom user message in the transcript

The admission and user transcript are written together, but later start/admission can still fail (skill blocked, oversized, binding failure). The cancellation only flips lifecycle state, not the transcript — a failed send remains visible and retries create duplicates.

[P2] Message reorder with identical content is rejected as a conflict

The metadata store compares target order byte-for-byte, so any non-trivial reorder is treated as a conflict. Existing tests pass only because reorders in those fixtures are no-ops.

[P2] Admission envelope can exceed storage limits undetected

Submitted and prepared payloads are checked separately, but the combined admission envelope (64 KiB limit) is not pre-validated. Inputs in the 32–49 KiB range pass early gates yet fail at admission, surfacing as an internal failure.

CI on 2ea21a689505 is test: success. These issues are independent of CI and require fixes before approval. Heads verified at time of review.

简体中文存在恢复路径与 transcript 残留等阻断问题,需修复后重审。

Preserve live Client capability bindings, make cancellation retries idempotent, and keep admission-backed transcripts out of compatibility Run synthesis until their root contract owns them.

Generated-by: Codex
@Astro-Han
Astro-Han force-pushed the fix/durable-message-lifecycle branch from 2ea21a6 to d691232 Compare August 24, 2026 19:29

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified a blocking issue in this head.

[P2] Retracted messages that never reached the Host remain visible in the transcript

When a queued Accepted message is retracted during reload, the lifecycle row is moved to cancelled but the session_messages user row and catalog preview inserted at admission stay visible. TUI expects still-queued text to return to the editor and only consumed messages to stay in history — after reload the old row is still paged as consumed steering, and resubmission leaves a duplicate.

Repro: next_turn submit → queue.retract → reopen. Lifecycle cancelled but readMessages still returns the user row with steeringEventId.

Fix: for accepted -> cancelled (never handed off) hide/remove the transcript row and recompute the preview with a lifecycle-aware projection; keep the row only for handed_off -> cancelled where Runtime actually consumed it. Cover reload + edit/resubmit.

Previous skill-recovery P2 is closed in this head. Checks on 490ffb31a4 are test: success.

简体中文撤回未送达消息仍留在历史视图,需区分未送达与已消费后取消。

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified the previous blocking finding still applies in this head.

[P2] Retracted Accepted messages remain visible

sqlite-session-metadata-store.ts:1605-1648 inserts the transcript row on Accepted, but cancelMessageAdmissions at :1935-1963 only flips lifecycle to cancelled without hiding rows that never reached the Host. After reload the never-handed-off row is still rendered, and resubmission leaves a duplicate.

Head f01aa233cd vs previous 490ffb31. Checks on f01aa233cd are test: success dependent on exact-head run.

简体中文该阻断在该头仍成立。

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update on 48b162326f:

The previous retracted-message P2 is now closed. Admission now commits a canonical draft only, and markMessagesHandedOff atomically materializes the transcript and advances lifecycle within a single SQLite transaction. Pre-handoff retract therefore leaves no transcript row to leak, while post-handoff cancel correctly retains history.

No remaining blocking issues. Previous Skill-recovery P2 remains closed.

Checks on 48b16232 are test: success.

简体中文该阻断已在该头闭合。

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update on 48b162326f: two blocking issues found in independent re-review.

[P2] Terminal root without provider-request proof leaves handed-off messages unsettled

A failed root before any model_call_attempt leaves handed_off rows in listUnsettledMessages() forever, causing O(N²) recovery scans on restart.

[P2] Settled admissions permanently retain a full message copy

After handoff, message_admissions keeps content_json (48 KiB) even though the transcript already holds the text, permanently doubling storage for executed messages.

Checks on 48b16232 are test: success, but these lifecycle issues remain independent of CI.

简体中文该头存在终态行为与存储双重阻断。

@M4n5ter M4n5ter left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

English

I found two blocking issues on exact head 48b162326f39ac1c83d7ebd8277a33ffd56ddba3: a failed terminal root can leave handed-off messages permanently unsettled, and every settled admission permanently retains a second full copy of the message payload. The hosted test check is green; I also ran 156 focused tests and four package builds, then reproduced both issues through the production coordinator/storage seams with real SQLite. The previous reorder and admission-envelope findings are closed on this head.

简体中文

我在 exact head 48b162326f39ac1c83d7ebd8277a33ffd56ddba3 上确认了两个阻断问题:失败终态的 root 可能让已 handoff 的消息永久处于未结算状态;每条已结算 admission 还会永久保留第二份完整消息 payload。托管 test 检查是绿色;我也运行了 156 个聚焦测试和四个 package build,并通过 production coordinator/storage seam 与真实 SQLite 复现了两个问题。此前的重排与 admission-envelope 问题在该 head 上已经闭合。

}
await this.#lifecycle.markMessagesExecuted(input.sessionId, executed);
}
if (input.terminalStatus !== 'cancelled') return;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

English

[P2] Please settle unproved sources when the root has reached a terminal state instead of returning here for every status except cancelled. After a message is handed off, a root can fail before any provider-request proof exists; with the production coordinator and a real SQLite lifecycle store, terminalStatus: 'failed' left the row in handed_off, still returned by listUnsettledMessageAdmissions, while its transcript row already existed. Restart recovery therefore processes the same terminal row forever; because recovery iterates N unsettled rows and each settlement lists all N again, stranded rows also create an O(N²) decode/scan path. Please give every terminal source a terminal lifecycle result (proved → executed, unproved → an explicit non-executed terminal state or cancelled) and cover failed-before-provider plus restart. Treat completed without proof as an invariant violation rather than leaving it unsettled.

简体中文

[P2] root 已进入终态时,不应对除 cancelled 外的所有状态都在这里直接返回,而需要终结没有 provider proof 的 source。消息 handoff 后,root 可能在任何 provider-request proof 出现前失败;使用 production coordinator 和真实 SQLite lifecycle store 复现时,terminalStatus: 'failed' 会让该 row 永久停在 handed_off,仍被 listUnsettledMessageAdmissions 返回,同时 transcript row 已经存在。重启恢复会永远重复处理这条终态 row;而 recovery 外层遍历 N 条 unsettled row、每次 settlement 又读取全部 N 条,因此还会形成 O(N²) 的 decode/scan 路径。请让每个 terminal source 都得到 terminal lifecycle 结果(proved → executed,unproved → 明确的 non-executed terminal state 或 cancelled),并补 failed-before-provider 与 restart 回归。completed 但没有 proof 应作为 invariant violation,而不是永久留在 unsettled。

const statement = this.db.prepare(
`
UPDATE message_admissions
SET lifecycle_state = ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

English

[P2] Please compact the admission payload when this transition makes the row terminal. This update changes only lifecycle_state, while the schema keeps content_json mandatory and no deletion/compaction path clears it. With a protocol-valid 48 KiB message, a real SQLite probe left an executed admission with 49,163 characters of content_json plus a separate 49,250-character transcript record. Normal long-lived Sessions therefore retain a second unused full-content copy for every settled message (roughly 469 MiB for 10,000 maximum-sized text messages before SQLite overhead), even though terminal APIs need only lifecycle identity. Please retain full content only while accepted, then atomically replace it with the minimal tombstone/digest at handoff, or split pending payloads from terminal lifecycle tombstones; add a settled-row retention assertion.

简体中文

[P2] 这次 transition 让 row 进入终态时,需要同时压缩 admission payload。当前更新只改变 lifecycle_state,schema 却要求 content_json 永远存在,也没有删除或 compact 路径。对一条协议允许的 48 KiB 消息做真实 SQLite 探针后,executed admission 仍保留 49,163 字符的 content_json,同时 transcript 中另有 49,250 字符记录。普通长寿命 Session 因此会为每条已结算消息永久保留第二份无消费者的完整正文;10,000 条最大文本约额外占用 469 MiB,且尚未计 SQLite overhead,而 terminal API 实际只需要 lifecycle identity。请只在 accepted 阶段保留完整内容,并在 handoff 时原子替换为最小 tombstone/digest,或把 pending payload 与 terminal lifecycle tombstone 分表;同时增加 settled-row retention 断言。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants